Method, device and system for generating a centerline for an object in an image

ABSTRACT

Systems and methods for generating a centerline for an object in an image are provided. An exemplary method includes receiving an image containing the object. The method also includes detecting at least one bifurcation of the object using a trained bifurcation learning network based on the image. The method further includes extracting the centerline of the object based on a constraint condition that the centerline passes through the detected bifurcation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 16/550,093, filed on Aug. 23, 2019, which claims the benefit ofpriority to U.S. Provisional Application No. 62/722,145, filed on Aug.23, 2018, the entire contents of both of which are incorporated hereinby reference.

TECHNICAL FIELD

The disclosure generally relates to medical image processing andanalysis. More specifically, this disclosure relates to a method, adevice, and a system for generating a centerline for an object, e.g., avessel, in an image.

BACKGROUND

Centerline is a type of skeleton representation of an object, with eachpoint equidistant to the object's boundary at a longitudinal position ofthe point. Centerline provides a concise representation that emphasizesgeometrical and topological properties of the object, such asconnectivity, length, direction, etc. especially for tree-like objects.It is widely used in optical character recognition, fingerprintrecognition, medical image (2D, 3D, or 4D) analysis, etc. example, inoptical character recognition task, correct extraction of thecenterlines of characters is essential to construct a robustrepresentation of characters in different fonts/sizes, etc. In medicalimage analysis of vessel tree structure, centerline extraction helpsimproving vessel segmentation and also enables the evaluation of vesseldiameter at each longitudinal location and detection/quantification ofstenosis, etc.

The current methods for extracting a centerline of an object,automatically or semi-automatically, may be divided into two majorcategories, morphological skeletonization and minimum cost path basedmethods. For morphological skeletonization methods such as erosion andthinning, small perturbations or noise on the image can easily lead tofalse positives of short centerlines (representing spurious branches).Although more global features may be adopted to reduce such falsepositives, however, even for moderately-sized images, usually hours oftime are required which is clinically unacceptable. For the minimum costpath based methods, users are required to specify explicitly geometricalconstraints, e.g., end points of unique branches and compute minimumcost paths. However, the current distance transform based cost imagecomputation and end points detection cannot handle objects with uneventhickness at different locations robustly, resulting in either falsepositive centerlines in thick regions or lack of centerline forthin/small branches.

Further, traditional distance cost images are often scale-variant, whichleads to inconsistent results between thick and thin regions. Andtraditional automated end point finding algorithms are often based onlocal maxima to detect protruding locations as end points. The end pointdetection usually struggles to balance between false positive detectionand missing end points.

Besides, traditional methods also fail in “kissing” cases when twobranches are partially close to each other, where the detectedcenterline for one branch can easily jump to the other branch. In orderto alleviate all the above shortcomings, some traditional methodsenforce a strong prior topological model such as a predefined number ofbranches with a predefined hierarchical structure. However, with suchrestrictions, the current methods are adapted only to very limitedapplications such as major airway center line extraction, whichobstructs its promotion and development.

The present disclosure is provided to overcome the technical defects inthe traditional method for extracting centerlines of object, with avariety of geometrical shapes and structures, in various images.

SUMMARY

In one aspect, a method for generating a centerline for an object isdisclosed. The method includes receiving an image containing the object.The method also includes detecting at least one bifurcation of theobject using a trained bifurcation learning network based on the image.Moreover, the method includes extracting the centerline of the objectbased on a constraint condition that the centerline passes through thedetected bifurcation.

In another aspect, a system for generating a centerline for an object isdisclosed. The system includes an interface configured to receive animage containing the object. The image is acquired by an imaging device.The system also includes a processor configured to detect at least onebifurcation of the object using a trained bifurcation learning networkbased on the image. In addition, the processor is configured to extractthe centerline of the object based on a constraint condition that thecenterline passes through the detected bifurcation.

In a further aspect, a device for generating a centerline for an objectin an image is disclosed. The device includes a detection unit and anextraction unit. The detection unit is configured to detect at least onebifurcation of the object, using a trained bifurcation learning networkbased on the image. The extraction unit is configured to extract thecenterline of the object based on a constraint condition that thecenterline passes through the detected bifurcation.

In yet another aspect, a non-transitory computer readable medium storinginstructions is disclosed. The instructions, when executed by aprocessor, perform a method for generating a centerline for an object.The method includes receiving an image containing the object. The methodalso includes detecting at least one bifurcation of the object using atrained bifurcation learning network based on the image. Moreover, themethod includes extracting the centerline of the object based on aconstraint condition that the centerline passes through the detectedbifurcation.

The method, device, system, and medium for generating a centerline foran object in an image have a better performance confronting complexsituations such “kissing” branches, crossing branches, etc., andincrease the detection rate and lower the false positive rate of the endpoints (and also the object) due to the much bigger model capacity andability of the learning network to learn from large amount of trainingdata.

It is to be understood that the foregoing general description and thefollowing detailed description are exemplary and explanatory only, andare not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, likereference numerals may describe similar components in different views.Like reference numerals having letter suffixes or different lettersuffixes may represent different instances of similar components. Thedrawings illustrate generally, by way of example, but not by way oflimitation, various embodiments, and together with the description andclaims, serve to explain the disclosed embodiments. Such embodiments aredemonstrative and not intended to be exhaustive or exclusive embodimentsof the present method, device, system, or non-transitory computerreadable medium having instructions thereon for implementing the method.

FIG. 1 illustrates a method for generating a centerline for an object inan image according to an embodiment of present disclosure;

FIG. 2 illustrates a centerline generation device and its workingprocess according to an embodiment of present disclosure;

FIG. 3 illustrates an example of the learning network (including a firstlearning network and a second learning network) used in the methodaccording to an embodiment of present disclosure;

FIG. 4 illustrates an example of the learning network (including thefirst learning network and the second learning network) used in themethod according to an embodiment of present disclosure;

FIG. 5 illustrates a training process of the learning network (includingthe first learning network and the second learning network) according toan embodiment of present disclosure;

FIG. 6 illustrates a training process of the learning network (includingthe first learning network and the second learning network) according toan embodiment of present disclosure;

FIG. 7 illustrates an example of the learning network used in the methodaccording to an embodiment of present disclosure; and

FIG. 8 depicts a block diagram illustrating an exemplary centerlinegeneration device, according to an embodiment of present disclosure.

DETAILED DESCRIPTION

Hereinafter, the technical term “object” is used as compared to thebackground of the image. For medical images, “object” may refer toorgans and tissues of interest, e.g., vessel, airway, glands. Foroptical character recognition, “object” may refer to characters. In someembodiments, medical image is used as an example of image and vessel isused as an example of the “object”, but the method, device, and systemin the embodiments may be easily and smoothly transformed to extractionof centerlines for other objects in other types of images. And thetechnical term “image” may refers to a complete image or an image patchcropped from the image.

FIG. 1 illustrates a computer-implemented method for generating acenterline for an object in an image according to an embodiment ofpresent disclosure. As shown in FIG. 1 , the centerline generationprocess 100 begins with receiving an image containing the objectacquired by an imaging device (step 101). The image may be a 2D image,3D image, or a 4D image. The image may be acquired directly by a variousof imaging modalities, such as but not limited to CT, digitalsubtraction angiography (DSA), Mill, functional MRI, dynamic contrastenhanced MRI diffusion MRI, spiral CT, cone beam computed tomography(CBCT), positron emission tomography (PET), single-photon emissioncomputed tomography (SPECT), X-ray imaging, optical tomography,fluorescence imaging, ultrasound imaging, radiotherapy portal imaging,or acquired by reconstruction based on the original images acquired bythe image device. For example, a 3D vessel volume image may be obtainedby reconstruction based on vessel DSA images at two different projectiondirections. The technical term “acquire” means any manner of obtaining,directly or indirectly, with or without additional image processing(noise reduction, cropping, reconstruction, etc.), and the acquiredimage is received as input image for the centerline generation process100.

Based on the input image (2D, 3D, or 4D image), a distance cost imagemay be automatically generated by a processor using a trained firstlearning network (step 102). The technical term “distance cost image”means an image with the same dimension as that of the input image andeach pixel thereon has an intensity indicating its distance from thecenterline. By means of the trained first learning network, thegeneration of the distance cost image is robust to complex situationssuch as “kissing” branches, due to the capacity and ability of the firstlearning network to learn from large amount of data covering complexsituations. Besides, the first learning network may adopt convolutionalnetwork, which has an efficient hierarchical learning capability, andmay have a better performance confronting complex situations such as“kissing” branches, crossing branches, etc.

In some embodiments, the ground truth distance cost image, which is usedfor training the first learning network, may be normalized so that theintensity of each of its pixels indicates a normalized distance of thepixel from the centerline of the object. The normalized distance of thepixel belonging to the object may be a ratio of its distance from thecenterline to the radius of the object at a longitudinal position ofcenterline corresponding to the pixel. In this manner, the intensity ofeach pixel belonging to the object may be 1 or less: the intensity ofthe pixel locating on the boundary of the object may be 1; and theintensities of the pixels within the boundary may be less than 1.Besides, the intensities of the pixels away from the object in theground truth distance cost image may be predefined at a certain value,such as but not limited to 1 or more. Through trained using thenormalized ground truth distance cost image as training data, the firstlearning network may learn from the scale-invariant distance cost of thepixels throughout the image, and may be used to efficiently compute ascale-invariant distance cost image, which may be robust to and mayhandle the objects with uneven thickness/diameters.

At step 103, the end points of the object may be detected by using atrained second learning network based on the input image. Although FIG.1 shows a sequence of step 102 and step 103, it does not intend to limitthe performing sequence of the two steps. Instead, as along as step 102and step 103 are performed after step 101 and before step 104, they maybe performed in any sequence. By means of the trained second learningnetwork, the detection rate may be significantly higher and the falsepositive rate may be significantly lower compared to the prior-artrule-based local maxima finding algorithm, due to the much bigger modelcapacity and ability of the second learning network to learn from largeamount of training data.

Then, the centerline generation process 100 may proceed to step 104: thecenterline of the objected may be extracted based on the distance costimage and the end points of the object as input. In some embodiments,for any two end points, a path connecting them with minimal distancecost (i.e., the sum of the distance cost along the path) may begenerated and the generated paths may be screened manually,semi-automatically, or automatically to filter out the false path (whichis inconsistent with the topological structure of the object, e.g., itstwo ending points are not topologically connected with each other inanatomical structure of the object). The minimal cost path may becomputed by various algorithms, such as but not limited to Dijkstra'salgorithm, A* search algorithm, Bellman-Ford algorithm, and fastmarching algorithm. Particularly, a path cost threshold may be presetand the path with a cost higher than the path cost threshold may beregarded as false path and cancelled.

In some embodiments, the detected end points may be screened and pairedfirstly, so as to find the starting end points and its correspondingterminating end points in pairs. For each pair of a starting end pointand its corresponding terminating end point, a path connecting them withminimal distance cost may be generated as a part of the centerline ofthe object. In this manner, the work load on false path generation (forthe end points which are not in pairs anatomically) may be saved and theextraction accuracy of the centerline may be further improved. Thepairing of the detected end points may be performed automatically,semi-automatically, or by manual intervention.

In some embodiments, the second learning network may adopt aconvolutional network, especially a fully convolutional network, so asto accelerate its computation on a GPU. Correspondingly, the secondlearning network may be used to predict an end point map based on theinput image, with intensity of each pixel of the end point mapindicating whether the pixel is an end point. Besides whether the pixelis an end point, the intensity may deliver richer information, such asbut not limited to whether the pixel is a starting end point, aterminating end point, or is not an ending point, with which end pointit is in pairs (if the pixel is an ending point), etc. Such informationmay be used to determine the primary topological structure of theobject, based on which the centerline extraction step (step 104) may beadapted (simplified) to improve the extraction accuracy, increasecomputation speed, and reduce (or even avoid) the manual intervention.

As an example, under a condition that a starting end point is in pairswith multiple terminal end points, which means that the centerlineassociated with them is tree-like shaped, a path with minimal distancecost connecting the starting end point and each terminal end point maybe determined and the determined paths may be integrated as thecenterline of the corresponding portion of the object defined by thestarting end point and the multiple terminal end points. Particularly,the starting end point serves as a root point of the tree-likecenterline and the every other terminal end points serves as a leaf ofthe tree-like centerline. The series of minimal cost paths mayconstitute the centerline of the object. In some embodiments, for thetree-like centerline, the paths with the same starting end points, ifare too close to each other (e.g., their distance is less than athreshold) in some part, the paths may be fused in said part to avoidfalse branches resulted from calculation error.

As another example, under a condition that a starting end point is inpairs with only one terminal end point, which means that the centerlineassociated with them is tube-like shaped, a path with minimal distancecost connecting them may be determined as the centerline of thecorresponding portion of the object defined by the starting end pointand the one terminal end point.

The centerline generation process 100 may avoid the spurious falsepositive centerlines generated by traditional morphological methods. Itdoes not depend on a predefined structure/topology of the target object,and may be applied widely to various objects with varying structures andtopologies.

In some embodiments, the centerline of the objected extracted at step104 may be a single pixel-wide line, which may deliver the geometricaland topological properties of the object with a high resolution andsensitivity.

FIG. 2 illustrates a centerline generation device 200 and its workingprocess according to an embodiment of present disclosure. As shown inFIG. 2 , centerline generation device 200 may include a generation unit201, a detection unit 202, and an extraction unit 203. The trained firstlearning network may be transmitted from a first training unit 204 tothe generation unit 201, and then the generation unit 201 may make useof the trained first learning network to generate a distance cost imagebased on the input image from the imaging device/medical image database209. The trained second learning network may be transmitted from asecond training unit 205 to the detection unit 202, and then thedetection unit 202 may make use of the trained second learning networkto detect the end points of the object based on the input image from theimaging device/medical image database 209. The distance cost imagegenerated by the generation unit 201 together with the end points of theobject detected by the detection unit 202 may be input into theextraction unit 203, which is configured to extract the centerline ofthe object based on the distance cost image with the end points of theobject as constraint conditions.

In some embodiments, the extraction unit 203 may comprise a pathgeneration unit 207 and an integration unit 208. The path generationunit 207 may be configured to generate a path connecting each pair of astarting end point and its corresponding terminating end point withminimal distance cost by means of any one of Dijkstra's algorithm, A*search algorithm, Bellman-Ford algorithm, and fast marching algorithm.And the integration unit 208 may be configured to integrate (e.g., add,fuse, etc.) all the generated paths as the centerline of the object.

In some embodiments, training process may be performed remote from thecenterline generation device 200 (as shown in FIG. 2 , neither of thefirst training unit 204 and the second training unit 205 is locatedwithin the centerline generation device 200) or performed locally at thecenterline generation device 200.

As shown in FIG. 2 , the first learning network and the training data(comprised by the medical image and its ground truth distance costimage, provided from the training data database 210) may be fed into thefirst training unit 204, so that the first training unit 204 may trainthe first learning network using the training data. In some embodiments,a normalization unit 206 may be added to perform a normalization on theground truth distance cost image, so as to provide ground truth costimage that is scale-invariant. Accordingly, the first learning networkmay learn about it and generate distance cost image that isscale-invariant. The second learning network and the training data(comprised by the medical image and its ground truth list of end points,provided from the training data database 210) may be fed to the secondtraining unit 205, so that the second training unit 205 may train thesecond learning network using the training data.

In some embodiments, a pairing and classification unit 211 may be addedon the upstream of the second training unit 205, so as to post-processthe end point labels in the medical image to provide ground truth endpoint map, with the intensity of its each pixel indicating whether thepixel is a starting end point or a terminating end point(classification) and with which end point it is in pairs (pairing). Theclassification may be performed by a variety of algorithms, such as butnot limited to identify the end point with a larger diameter of theobject at its position in the longitudinal direction of the object asstarting end point. And the pairing may be performed by e.g., referringto the topological connection relationship between the end points.Particularly, if there is a connection portion of the object coveringthe two end points, then the two end points may be identified to be inpairs with each other. By means of the pairing and classification unit211, the trained second learning network may predict directly an endpoint map, with the intensity of its each pixel indicating whether thepixel is a starting end point or a terminating end point and with whichend point it is in pairs. As an alternative option, a pairing unit 212and a starting end point selection unit 213 may be added into theextraction unit 203. The pairing unit 212 may be configured to pair thedetected end points and the starting end point selection unit 213 may beconfigured to select a subset of the detected end points as starting endpoints. And the remained end points may be then identified asterminating end points. Correspondingly, the detection unit 202 maycomprise a prediction unit (not shown), and the prediction unit may beconfigured to predict an end point map using the trained second learningnetwork based on the image, with intensity of each pixel of the endpoint map indicating whether the pixel is an end point.

The method according to an embodiment of present disclosure makes use ofa learning network, comprising a first learning network and a secondlearning network. As shown in FIG. 3 , the first learning network 308may comprise an encoder 302 and a decoder 303, and the second learningnetwork 309 may comprise an encoder 304 and a decoder 305. The encoders302, 304 may be configured to extract features from the input image 301,the decoder 303 may be configured to generate the distance cost image306 based on the features extracted by the encoder 302, and the decoder305 may be configured to detect the end points 307 of the vessel basedon the features extracted by the encoder 304. In some embodiments, eachof the first learning network 308 and the second learning network 309may be constructed based on convolutional network, which may consist ofhierarchical combinations of convolutional layer, pooling layer, andup-sampling layer, etc. For example, the convolutional network may beimplemented by any one of VGG, ResNet, DenseNet convolutional networks,etc.

As shown in FIG. 3 , the first learning network 308 and the secondlearning network 309 are independent from each other, and thus may betrained independently from each other.

FIG. 4 illustrates another example of the learning network used in themethod according to an embodiment of present disclosure. The learningnetwork 400 differs from the learning network 300 only in that the firstlearning network 408 shares an encoder 402 with the second learningnetwork 409. The decoders 403, 405, the input image 401 the distancecost image 406, and the end points 407 of the vessel are each similar tothe decoders 303, 305, the input image 301, the distance cost image 306,and the end points 307 of the vessel in FIG. 3 , and thus theirconfigurations are omitted here. The construction of the learningnetwork as shown in FIG. 4 is significantly simplified compared that asshown in FIG. 3 , much less parameters of the learning network need tobe determined, and thus both the training of and the prediction usingthe learning network as shown in FIG. 4 may be easier and accelerated.

In some embodiments, the first learning network and the second learningnetwork as shown in FIG. 3 may be trained separately or in an integratedmanner. As shown in 5, the training process 500 may start at step 501 ofloading a piece (or batch) of a first training data for the firstlearning network and a second training data for the second learningnetwork. The first training data may be comprised of the input vesselimage and the corresponding ground truth distance cost image, and thesecond training data may be comprised of the input vessel image and thecorresponding ground truth end point map.

At step 502, a first loss function may be calculated based on the firsttraining data using the current parameters of the first learningnetwork, and a second loss function may be calculated based on thesecond training data using the current parameters of the second learningnetwork. The first loss function and the second loss function may beintegrated (e.g., but not limited to weighted mean squared error and/orcross entropy, etc.) at step 503.

At step 504, the parameters of the first and second learning network maybe adjusted based on the integrated loss function. Then it may determinewhether there is still other piece (batch) of training data (step 505),if so, the process 500 may proceed back to step 501, otherwise theprocess 500 ends. By means of the integrated loss function, the trainedfirst and second learning network, as a whole, may provide niceperformance on both distance cost image and end point map predictions,which serve as the base for the subsequent minimum cost path generation.

In some embodiments, the training process 500 may be slightly adjust tobe applied to the learning network as shown in FIG. 4 , wherein thefirst learning network shares the encoder with the second learningnetwork. As an example, at step 504, parameters of two decoders (of thefirst and second learning network) and that of only one encoder (i.e.,the common encoder of the first and second learning network) need to beadjusted based on the integrated loss function, and thus the trainingprocess may be simplified and accelerated.

FIG. 6 illustrates a training process 600 of the learning network asshown in FIG. 6 according to an embodiment of present disclosure,wherein the steps 601-605 are similar to steps 501-505. Compared withthe training process 500 as shown in FIG. 5 , a pre-training of thefirst learning network (steps 6011-6014) is added preceding the step601. At step 6011, a piece of the first training data may be loaded. Thefirst loss function may be determined based on the first training dataat step 6012, and then the parameters of the first learning network maybe adjusted based on the first loss function at step 6013. Then it maydetermine whether there is still other piece of the first training datafor pre-training of the first learning network (step 6014), if so, theprocess 600 may proceed back to step 6011, otherwise the process 600 mayproceed to the step 601 to perform the integrated training of the firstand second learning network. Since the ground truth distance cost imagesare easier to obtain and the intensity profile of the pixels of theground truth distance cost image is relatively dense (compared to theintensity profile of the pixels of the ground truth, wherein a minorityof intensities are non-zero since the sparse distribution of the endpoints in the vessel image), the training of the first learning networkmay be much easier and quicker. In this manner, the parameters of theencoder of the pre-trained first learning network may be used as initialparameters of the common encoder to perform the parameter adjusting step604. It turns out that the integrated training of the first and secondlearning network using the parameters of pre-trained encoder as initialparameters of the common encoder may be significantly acceleratedcompared to that using arbitrary predefined initial parameters.

In some embodiments, not all the pieces of the first training data needto be adopted for the pre-training of the first learning network.Instead, a subset thereof may be adopted for the pre-training of thefirst learning network, so that the whole training process 600 may befurther accelerated.

In some embodiments, the ground truth end point map for training thesecond learning network may be obtained by setting the intensities ofthe pixels in an area around each end point based on the intensity ofthe pixel at the end point. As example, intensities of 5-10 pixelsaround each end point may be set to non-zero values. In this manner, thepositive samples in the ground truth end point map may be significantlyincreased, so as to alleviate the unbalance between the positive andnegative samples (due to the sparse distribution of the end points inthe vessel) and thus the training of the second learning network maybecome much easier.

Modified example of the learning network according to any embodiment ofpresent disclose may be also adopted. As shown in FIG. 7 , the learningnetwork 700 may comprise the first learning network, the second learningnetwork, and a third learning network. The first learning network maycomprise: an encoder 702, which is configured to extract features fromthe input image; a decoder 703, which is configured to generate adistance cost image 706 based on the extracted features. The secondlearning network may comprise: an encoder 704, which is configured toextract features from the input image; a decoder 705, which isconfigured to generate an end point map 707 based on the extractedfeatures. And the third learning network may comprise: an encoder 709,which is configured to extract features from the input image; a decoder710, which is configured to detect the bifurcations of the object 711based on the extracted features. Then the distance cost image 706, theend point map 707, and the bifurcations 711 may be fed together into aminimum cost path extractor 712, so that the minimum cost path extractor712 may extract the minimum cost path connecting the end points as thecenterline of the object based on the distance cost image 706 and theend point map 707 under a constraint condition that the centerline passthrough the detected bifurcations 711. In this manner, the falsebranches of centerline may be efficiently avoided. In some embodiments,the first, second, and the third learning network may be constructed byconvolutional network.

In some embodiments, one or more attention unit 708 may be added to anyone of the encoders 702, 704, and 709 at location(s) therein so as toincrease weights of the features extracted at the location correspondingto the object compared to that of the features not corresponding to theobject. In this manner, the respective learning network may focus moreprecisely to regions of interest (e.g., the regions at the locationcorresponding to the object).

FIG. 8 illustrates a block diagram of an exemplary centerline generationsystem 800 according to an embodiment of present disclosure. Thecenterline generation system 800 may include a network interface 807, bymeans of which the centerline generation system 800 (or the centerlinegeneration device therein, which refers to the other components than thenetwork interface 807) may be connected to the network (not shown), suchas but not limited to the local area network in the hospital or theInternet. The network can connect the centerline generation system 800with external devices such as an image acquisition device (not shown),medical image database 808, and an image data storage device 809. Animage acquisition device may use any type of imaging modalities, such asbut not limited to CT, digital subtraction angiography (DSA), MRI,functional MRI, dynamic contrast enhanced—MRI, diffusion MRI, spiral CT,cone beam computed tomography (CBCT), positron emission tomography(PET), single-photon emission computed tomography (SPECT), X-ray,optical tomography, fluorescence imaging, ultrasound imaging,radiotherapy portal imaging.

In some embodiments, the centerline generation system 800 may be adedicated intelligent device or a general-purpose intelligent device.For example, the system 800 may adopt a computer customized for imagedata acquisition and image data processing tasks, or a server placed inthe cloud. For example, the system 800 may be integrated into the imageacquisition device.

The centerline generation system 800 may include an image processor 801and a memory 804, and may additionally include at least one of aninput/output 802 and an image display 803.

The image processor 801 may be a processing device that includes one ormore general processing devices, such as a microprocessor, a centralprocessing unit (CPU), a graphics processing unit ((IPU), and the like.More specifically, the image processor 801 may be a complex instructionset computing (CIBC) microprocessor, a reduced instruction set computing(RISC) microprocessor, a very long instruction word (VLIW)microprocessor, a processor running other instruction sets, or aprocessor that runs a combination of instruction sets. The imageprocessor 801 may also be one or more dedicated processing devices suchas application specific integrated circuits (ASICs), field programmablegate arrays (FPGAs), digital signal processors (DSPs), system-on-chip(SoCs), and the like. As would be appreciated by those skilled in theart, in some embodiments, the image processor 801 may be aspecial-purpose processor, rather than a general-purpose processor. Theimage processor 801 may include one or more known processing devices,such as a microprocessor from the Pentium™, Core™, Xeon™, or Itanium®family manufactured by Intel™, the Turion™, Athlon™, Sempron™ Opteron™,FX™, Phenom™ family manufactured by AMD™, or any of various processorsmanufactured by Sun Microsystems. The image processor 801 may alsoinclude graphical processing units such as a GPU from the GeForce®;Quadro®, Testa® family manufactured by Nvidia™ GMA, Iris™ familymanufactured by Intel™, or the Radeon™ family manufactured by AMD™. Theimage processor 801 may also include accelerated processing units suchas the Desktop A-4 (6, 6) Series manufactured by AMD™, the Xeon Phi™family manufactured by Intern™. The disclosed embodiments are notlimited to any type of processor(s) or processor circuits otherwiseconfigured to meet the computing demands of identifying, analyzing,maintaining, generating, and/or providing large amounts of imaging dataor manipulating such imaging data to generate a distance cost imageusing a trained first learning network based on the input image, detectthe end points of the object using a trained second learning networkbased on the input image, generate a minimum cost path connecting theend points, integrate the series of generated minimum cost paths, and/ortrain the learning network, or to manipulate any other type of dataconsistent with the disclosed embodiments. In addition, the term“processor” or “image processor” may include more than one processor,for example, a multi-core design or a plurality of processors eachhaving a multi-core design. The image processor 801 can executesequences of computer program instructions, stored in memory 804, toperform various operations, processes, methods disclosed herein.

The image processor 801 may be communicatively coupled to the memory 804and configured to execute computer-executable instructions storedtherein. The memory 804 may include a read only memory (ROM), a flashmemory, random access memory (RAM), a dynamic random-access memory(DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM, a static memory(e.g., flash memory, static random access memory), etc., on whichcomputer executable instructions are stored in any format. In someembodiments, the memory 804 may store computer-executable instructionsof one or more image processing program(s) 805. The computer programinstructions can be accessed by the image processor 801, read from theROM, or any other suitable memory location, and loaded in the RAM forexecution by the image processor 801. For example, memory 804 may storeone or more software applications. Software applications stored in thememory 804 may include, for example, an operating system (not shown) forcommon computer systems as well as for soft-controlled devices.

Further, memory 804 may store an entire software application or only apart of a software application (e.g. the image processing program (s)805) to be executable by the image processor 801. In addition, thememory 804 may store a plurality of software modules, for implementingthe respective steps of the method for generating a centerline for anobject in an image or the process for training the learning networkconsistent with the present disclosure. For example, the first trainingunit 204, the normalization unit 206, the generation unit 201, thesecond training unit 205, the pairing and classification unit 211, thedetection unit 202, the extraction unit 203, the path generation unit207, the integration unit 208, the pairing unit 212, and the startingend point selection unit 213 (as shown in FIG. 2 ), may be implementedas soft modules stored on the memory 804, especially as image processprogram(s) 805. For another example, at least the generation unit 201,the detection unit 202, and the extraction unit 203 are implemented assoft modules (e.g. the image processing program(s) 805) stored on thememory 804 the first training unit 204 and the second training unit 205,as well as the normalization unit 206 and the pairing and classificationunit 211, may be located remote from the centerline generation system800 and communicate with the generation unit 201/detection unit 202 toenable it receive the trained corresponding learning network, which isalready trained by the first and second training unit 204, 205 with thetraining data from the training data database 210 (in an off-linetraining process) and/or the fresh training data (i.e., the distancecost image automatically generated and the end points automaticallydetected therefrom together with the corresponding input image) from thegeneration unit 201 and the detection unit 202 (in an on-line trainingprocess), so as to generate the centerline of the object in the inputimage.

Besides, the memory 804 may store data generated/buffered when acomputer program is executed, for example, medical image data 806,including the medical images transmitted from image acquisitiondevice(s), medical image database 808, image data storage device 809,etc. In some embodiments, medical image data 806 may include theimage(s) received from the image acquisition devices to be treated bythe image processing program(s) 805, and may include the medical imagedata generated during performing the method of generating the centerlineof the object and/or training the learning network(s).

Besides, the image processor 801 may execute the image processingprogram(s) 805 to implement a method for generating centerline of theobject, then associate the input image with the corresponding distancecost image automatically generated and the end points automaticallydetected, and transmit the same into the memory 804, especially as themedical image data 806 therein. In this manner, each on-line centerlinegeneration process may generate a piece of fresh training data to updatethe medical image data 806. By means of executing the first and secondtraining unit 204, 205 as shown in FIG. 2 , the image processor 801 maytrain the first and second learning networks in an on-line manner toupdate the existing parameters (such as the weights) in the currentlearning network. In some embodiments, the updated parameters of thetrained learning network may be stored in the medical image data 806,which may then be used in the next centerline generation for the sameobject of the same patient. Therefore, if the image processor 801determines that the centerline generation system 800 has performed acenterline generation for the same object of the present patient, thenthe latest updated learning networks for centerline generation may berecalled and used directly.

In some embodiments, the image processor 801, upon performing an on-linecenterline generation process, may associate the input image togetherwith the automatically (or semi-automatically) generated centerline ofthe object as medical image data 806 for presenting and/or transmitting.In some embodiments, the input image together with the generatedcenterline may be displayed on the image display 803 for the user'sreview. In some embodiments, the medical image data by associating theinput image with the generated centerlines (or the distance cost imageand the end points) may be transmitted to the medical image database808, so as to be accessed, obtained, and utilized by other medicaldevices, if needed.

In some embodiments, the image data storage device 809 may be providedto exchange image data with the medical image database 808, and thememory 804 may communicate with the medical image database 808 to obtainthe images of the current patient. For example, the image data storagedevice 809 may reside in other medical image acquisition devices, e.g.,a CT which performs scan on the patients. The slices of the patients onthe object (such as vessel) may be transmitted, reconstructed into avolumetric image and saved into the medical image database 808, and thecenterline generation system 800 may retrieve the volumetric image ofthe object from the medical image database 808 and generate centerlinefor the object in the volumetric image.

In some embodiments, the memory 804 may communicate with the medicalimage database 808 to transmit and save the input volumetric imageassociated with the generated distance cost image and the detected endpoints into the medical image database 808 as a piece of training data,which may be used for off-line training as described above.

For example, the image display 803 may be an LCD, a CRT, or an LEDdisplay.

The input/output 802 may be configured to allow the centerlinegeneration system 800 to receive and/or send data. The input/output 802may include one or more digital and/or analog communication devices thatallow the system 800 to communicate with a user or other machine anddevice. For example, the input/output 802 may include a keyboard and amouse that allow the user to provide an input.

In some embodiments, the image display 803 may present a user interface,so that the user, by means of the input/output 802 together with theuser interface, may conveniently and intuitively correct (such as edit,move, modify, etc.) the automatically generated centerline of theobject, the automatically generated distance cost image, and theautomatically detected end points.

The network interface 807 may include a network adapter, a cableconnector, a serial connector, a USB connector, a parallel connector, ahigh-speed data transmission adapter such as optical fiber, USB 6.0,lightning, a wireless network adapter such as a Wi-hi adapter, atelecommunication (6G, 4G/LTE, etc.) adapters. The system 800 may beconnected to the network through the network interface 807. The networkmay provide the functionality of local area network (LAN), a wirelessnetwork, a cloud computing environment (e.g., software as a service,platform as a service, infrastructure as a service, etc.), aclient-server, a wide area network (WAN), and the like.

Various operations or functions are described herein, which may beimplemented as software code or instructions or defined as software codeor instructions. Such content may be source code or differential code(“delta” or “patch” code) that can be executed directly (“object” or“executable” form). The software code or instructions may be stored incomputer readable storage medium, and when executed, may cause a machineto perform the described functions or operations and include anymechanism for storing information in the form accessible by a machine(e.g., computing device, electronic system, etc.), such as recordable ornon-recordable media (e.g., read-only memory (ROM) random access memory(RAM), disk storage media, optical storage media, flash memory devices,etc.).

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations of theembodiments will be apparent from consideration of the specification andpractice of the disclosed embodiments.

Exemplary methods described herein can be machine orcomputer-implemented at least in part. Some examples can include anon-transitory computer-readable medium or machine-readable mediumencoded with instructions operable to configure an electronic device toperform methods as described in the above examples. An implementation ofsuch methods can include software code, such as microcode, assemblylanguage code, a higher-level language code, or the like. The variousprograms or program modules can be created using a variety of softwareprogramming techniques. For example, program sections or program modulescan be designed in or by means of Java, Python, C, C++, assemblylanguage, or any known programming languages. One or more of suchsoftware sections or modules can be integrated into a computer systemand/or computer-readable media. Such software code can include computerreadable instructions for performing various methods. The software codemay form portions of computer program products or computer programmodules. Further, in an example, the software code can be tangiblystored on one or more volatile, non-transitory, or non-volatile tangiblecomputer-readable media, such as during execution or at other times.Examples of these tangible computer-readable media can include, but arenot limited to, hard disks, removable magnetic disks, removable opticaldisks (e.g., compact disks and digital video disks), magnetic cassettes,memory cards or sticks, random access memories (RAMs), read onlymemories (ROMs), and the like.

Moreover, while illustrative embodiments have been described herein, thescope includes any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations or alterations based on the presentdisclosure. The elements in the claims are to be interpreted broadlybased on the language employed in the claims and not limited to examplesdescribed in the present specification or during the prosecution of theapplication. Further, the steps of the disclosed methods can be modifiedin any manner, including by reordering steps or inserting or deletingsteps. It is intended, therefore, that the descriptions be considered asexamples only, with a true scope being indicated by the following claimsand their full scope of equivalents.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with each other. Otherembodiments can be used, such as by one of ordinary skill in the artupon reviewing the above description. Also, in the above DetailedDescription, various features may be grouped together to streamline thedisclosure. This should not be interpreted as intending that anunclaimed disclosed feature is essential to any claim. Thus, thefollowing claims are hereby incorporated into the Detailed Descriptionas examples or embodiments, with each claim standing on its own as aseparate embodiment, and it is contemplated that such embodiments can becombined with each other in various combinations or permutations. Thescope of the invention should be determined with reference to theappended claims, along with the full scope of equivalents to which suchclaims are entitled.

The invention claimed is:
 1. A method for generating a centerline of anobject, comprising: receiving an image containing the object, whereinthe image is acquired by an imaging device; detecting, by a processor,at least one bifurcation of the object using a trained bifurcationlearning network based on the image; detecting, by the processor, endpoints of the object; and extracting, by the processor, the centerlineof the object based on the end points of the object and a constraintcondition that the centerline passes through the detected bifurcation.2. The method of claim 1, wherein the method further comprisesgenerating, by the processor, a distance cost image using a traineddistance cost learning network based on the image, wherein theextracting is performed further based on the distance cost image.
 3. Themethod of claim 2, wherein an intensity of each pixel of the distancecost image is indicative of a distance of the pixel from the centerlineof the object.
 4. The method of claim 1, wherein the end points of theobject are detected using a trained end point learning network based onthe image.
 5. The method of claim 4, wherein the detecting the endpoints of the object further comprises: predicting, by the processor, anend point map using the end point learning network based on the image,wherein an intensity of a pixel of the end point map is indicative ofwhether the pixel corresponds to a starting end point or a terminatingend point, or the pixel does not correspond to an ending point.
 6. Themethod of claim 4, wherein the extracting the centerline of the objectcomprises: generating a path connecting a pair of end points with aminimal distance cost, wherein the path passes through the detectedbifurcation.
 7. The method of claim 1, wherein the bifurcation learningnetwork is a convolutional network.
 8. The method of claim 1, whereinthe extracted centerline of the object comprises a single pixel-wideline.
 9. The method of claim 1, wherein the bifurcation learning networkcomprises an encoder configured to extract features, and the encoderfurther comprises an attention unit applying larger weights on featuresextracted at a location corresponding to the object compared to weightson features extracted at another location not corresponding to theobject.
 10. The method of claim 1, wherein the bifurcation learningnetwork comprises an encoder configured to extract features and adecoder configured to detect the bifurcations of the object based on theextracted features.
 11. The method of claim 1, wherein the objectcomprises a vessel.
 12. A system for generating a centerline of anobject, comprising: an interface configured to receive an imagecontaining the object, wherein the image is acquired by an imagingdevice; and a processor configured to: detect at least one bifurcationof the object using a trained bifurcation learning network based on theimage; detect end points of the object; and extract the centerline ofthe object based on the end points of the object and a constraintcondition that the centerline passes through the detected bifurcation.13. The system of claim 12, wherein the processor is further configuredto generate a distance cost image using a trained distance cost learningnetwork based on the image, wherein the extracting is performed furtherbased on the distance cost image.
 14. The system of claim 12, whereinthe end points of the object are detected using a trained end pointlearning network based on the image.
 15. The system of claim 14, whereinthe processor is further configured to: generate a path connecting apair of a starting end point and a corresponding terminating end pointwith a minimal distance cost, wherein the path passes through thedetected bifurcation.
 16. The system of claim 12, wherein thebifurcation learning network is a convolutional network.
 17. The systemof claim 12, wherein the extracted centerline of the object comprises asingle pixel-wide line.
 18. The system of claim 12, wherein thebifurcation learning network comprises an encoder configured to extractfeatures, and the encoder further comprises an attention unit applyinglarger weights on features extracted at a location corresponding to theobject compared to weights on features extracted at another location notcorresponding to the object.
 19. The system of claim 12, wherein theobject comprises a vessel.
 20. A non-transitory computer readable mediumstoring instructions that, when executed by a processor, perform amethod for generating a centerline of an object, the method comprising:receiving an image containing the object, wherein the image is acquiredby an imaging device; detecting at least one bifurcation of the objectusing a trained bifurcation learning network based on the image;detecting end points of the object; and extracting the centerline of theobject based on the end points of the object and a constraint conditionthat the centerline passes through the detected bifurcation.